Add LoRA-MPO integration for enhanced parameter efficiency #2868
Conversation
This PR introduces Matrix Product Operator (MPO) integration with LoRA to improve parameter efficiency and training stability. Key changes: - Implement lorampo_init method in LoraLayer for MPO-based initialization - Add lora_mpo configuration option to LoraConfig - Update training scripts and utilities to support MPO-LoRA - Add example training script for MPO-LoRA experiments Features: - Integration with existing LoRA infrastructure - Support for MPO-based weight initialization - Backward compatibility with standard LoRA This enhancement allows users to leverage MPO decomposition for more efficient parameter adaptation while maintaining the simplicity of LoRA usage.
- Resolved conflict in src/peft/tuners/lora/config.py - Kept both lora_mpo and ensure_weight_tying fields - Integrated all main branch updates including new methods (DeLoRA, OSF, WaveFT) - Updated CI workflows and documentation
|
For your convenience, here are some friendly points to help with a quick check. (1) How to quickly test this PRYou can directly test the Simply modify the following two lines: (2) What tests have been doneWe validated the PR under the following setting:
Ours results:
Observation: Compared to lora, |
|
Hi, could a maintainer please approve and run the pending workflows for this PR? They’re currently blocked with “2 workflows awaiting approval”. Thanks! |
There was a problem hiding this comment.
Thanks for this PR to add MPO to PEFT. Just for clarification, this is the corresponding paper and you are the first author, is that right?
I read some parts of the paper and although the mathematics are a bit outside my area of expertise, I think I get the general idea. However, what is unclear to me is how this approach relates to LoRA. This implementation is basically a small variation to LoRA, with a different initialization and a different forward path. But the paper (because of its age) doesn't mention LoRA at all. Is there a follow up paper to explain the relationship or could you please explain here how it's possible to express MPO in terms of LoRA?
Since this method is a LoRA variant, I would like you to implement it in terms of a LoraVariant subclass. To see how that works, check for instance this PR.
Moreover, before we can proceed, please make the following changes:
- The initialization has a dependency on
matrix2mpo_plus. However, we don't want to add another dependency to PEFT, especially not to small packages like this. Would it be possible to include the required code intompo_shape_calculator.py(let's rename it tompo_utils.pythen), aka to vendor the code? - There are a few comments in Chinese, could you please translate to English?
- Regarding the included example: Let's not edit the existing one but rather add a separate example.
2 participants
This is a PR for parameter-efficient fine-tuning method, MPOP[1], which introduces Matrix Product Operator (MPO) integration with LoRA (we call
lorampohere) to improve parameter efficiency and training stability.Key changes:
lorampomethod in MLP layers using MPO-based initializationlorampolorampoexperimentsFeatures:
This enhancement allows users to leverage MPO decomposition for more efficient parameter adaptation while maintaining the simplicity of LoRA usage.
[1] Liu et al. Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators. ACL 2021